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INTRODUCTION 


This  study  was  undertaken  as  jpart  of  a  continuing  series  of 
studies  designed  to  validate  the  Human  Operator  Simulator ^ (HOS) .  HOS 
is  intended  to  be  used  as  a  system  evaluation  tool  that  can  be  applied 
to  a  detailed  system  design  priori  to  the  development  of  a  hardware 
prototype.  Consequently,  an  assessment  of  its  ability  to  model  operator 
workload  problems  realistically  was  considered  to  be  of  prime  importance. 
The  study  was  designed  to  show  that^fkos,  a  general-purpose  model  of 
human  performance,  can  accurately  simulate  the  kinds  of  complex  inter¬ 
actions  between  competing  task  demands  experienced  by  a  human  operator 
performing  a  complex  mission. 

Other  analytic  approaches  to  the  modeling  of  task  workload 
have  been  developed  (e.g.,  Siegel  and  Wolf,  1962;  Murphy  and  Gurman, 

1972;  Linton,  Jahns,  and  Ch^itelier,  1977)  but  they  all  require  the  user 
to  supply  explicit  estimates  of  task  demands  in  terms  of  processing 
time  and  channel  requirements.^  HOS,  on  the  other  hand,  requires  that 
the  user  supply  only  fairly  objective  information  about  hardware  system 
configuration  and  dynamics,  operator  size  and  general  performance 
characteristics,  and  sequences  of  task  operations. 


^-In  this  study,  an  operator  performing  both  a  primary  pursuit 
tracking  task  and  a  "neutral”  secondary  task  was  simulated.  The  tracking 
task  was  adopted  as  the  primary  task  because  of  the  following: 

fj  It  is  commonly  encountered  by  operators  of  modern  control 
systems . 

^  Its  continuous  nature  ensures  that  it  will  be  sensitive 
to  interruptions  by  a  secondary  task. 

3;  i  It  perm ! ts  definitions  of  continuously  variable  performance 
measures. 


The  secondary  task  is  neutral  in  the  sense  that  the  operator 
"does  nothing,"  thereby  avoiding  the  task  interactions  that  occur  when 
two  tasks  compete  for  processing  channels.  The  secondary  task  was 
am  externally  scheduled  interruption  of  the  tracking  task.  Experimentally, 
this  could  be  achieved  by,  for  example,  blanking  out  the  display  so  that 
an  operator  engaged  in  visually  monitored  pursuit  tracking  could  not 
see  the  display  for  a  specified  length  of  time.  _The  study  examined  how 
primary  task  performance  was  influenced  by  the  schedule  of  interruptions 
and  other  parameters  that  affect  the  difficulty/  oX  the  tracking  task 
(i.e.,  signal  characteristics  and  the  operator’s  criterion  of  acceptable 
performance) . 
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2. 


MODELS  FOR  MANUAL  PURSUIT  TRACKING 


While  a  simple  model  could  be  constructed  that  would  enable  HOS 
to  perform  a  simple  tracking  function,  our  objective  was  to  devise  a 
model  that  would  embody  the  kind  of  adaptive  characteristics  that  human 
operators  possess.  In  this  section,  we  will  discuss  the  rationale 
behind  the  particular  model  that  was  used,  the  model  itself,  and  the 
simulation  studies  that  were  performed  to  determine  some  general  per¬ 
formance  features  of  the  model.  Section  3  discusses  studies  in  which 
the  completed  tracking  model  was  used  with  a  variety  of  interruption 
schedules  to  address  some  basic  workload  measurement  issues. 

2 . 1  Control  Theory  Models 

Most  investigators  who  have  modeled  tracking  behavior  have 
adopted  the  "black  box"  approach  characteristic  of  mathematical  control 
theory.  These  control  theory  models  have  resulted  in  some  significant 
insights  into  the  organization  of  skilled  human  performance.  They  can 
achieve  exceptionally  accurate  predictions  of  performance  on  most  track¬ 
ing  tasks  and  can  account  for  the  systematic  changes  in  performance  when 
some  interesting  task  factors  are  varied  (e.g.,  the  dynamics  of  the 
control-display  relationship) .  However,  they  generally  explicitly 
disregard  the  operator's  internal  processing  structure,  and  often  fail 
to  provide  a  sufficiently  detailed  description  of  performance  components 
to  enable  reliable  predictions  of  tracking  performance  to  be  made  in 
many  situations  of  interest.  For  example,  they  do  not  permit  the 
identification  of  tasks  that  could  be  performed  concurrently  without 
compromising  performance  on  the  tracking  task. 

In  general,  the  best  tracking  model  would  be  one  whose  param¬ 
eter  values  could  be  determined  completely  from  observable  characteris¬ 
tics  of  the  tracking  apparatus  and  the  operator's  basic  performance 
characteristics  (e.g.,  reaction  time,  fatigue  level).  Unfortunately, 
control  theory  models  generally  do  not  admit  these  possibilities. 

Rather,  the  control  theory  approach  fits  a  general  model  to  a  specific 
task  situation  by  estimating  all  free  parameters  in  order  to  best  fit 
the  model  to  actual  performance.  Such  a  modeling  strategy  is  clearly 
inapplicable  to  the  prediction  of  operator  performance  with  a  proposed 
hardware  system  before  a  prototype  system  can  be  constructed,  expecially 
if  the  proposed  system  is  very  different  from  other  systems  for  which 
performance  data  is  available. 

2 . 2  Information  Processing 


A  number  of  investigators  have  studied  the  cognitive  and  motor 
components  of  tracking  performance  and  some  fairly  comprehensive  con¬ 
ceptual  models  have  been  advanced  (e.g..  Crossman,  1960).  But,  to 
date,  there  has  been  a  singular  absence  in  the  published  literature 
of  detailed  models  on  the  operation  of  the  processing  components  that 
contribute  to  tracking  performance.  The  two  factors  that  are  probably 
most  responsible  for  this  are: 


(1)  Most  human  information  processing  operations  are  generally 
understood  as  discrete  events  (e.g.,  detection,  decision, 
response) ,  while  tracking  performance  is  apparently  con¬ 
tinuous. 

(2)  People  are  generally  recognized  to  be  extremely  flexible 
information  processors. 

The  problem  raised  by  the  apparent  continuity  of  tracking  per¬ 
formance  in  many  situations  and  the  presumed  discreteness  of  human  per¬ 
formance  components,  can  be  resolved  by  the  modeler  in  a  number  of  ways. 
One  way  is  to  question  the  continuous  nature  of  tracking  performance 
on  theoretical  or  empirical  grounds.  Hick  (1948) ,  for  example,  has 
argued  that  the  refractory  period  of  neural  processes  must  force  the 
human  operator  to  act  as  a  discontinuous  controller.  Indeed,  frequent, 
abrupt  control  movements  have  been  observed  even  when  the  tracked 
signal  is  slow  and  fluid  (North,  Lomnicki,  and  Zaremba,  1948).  Some 
psychologists  have  suggested  that  all  human  perceptual  processing  can 
occur  only  as  quantitative  events  that  are  paced  by  a  rapid  biological 
cycle  (Stroud,  1955;  Kristof ferson,  1967) .  Though  compelling  evidence 
against  this  theory  of  the  "psychological  moment"  has  been  raised  by 
experiments  reported  by  Allport  (1968)  and  Baron  (1971) ,  there  are 
still  clear  limitations  in  the  rates  at  which  perceptual  and  motor 
events  can  proceed.  Fitts  and  Posner  (1967)  suggest  that  the  number 
of  repetitive  stimuli  that  can  be  perceptually  distinguished  and  the 
frequency  at  which  repetitive  motor  processes  c~*n  be  executed  are  both 
limited  to  about  ten  events  per  second.  Of  course,  people  are  quite 
capable  of  executing  complex  continuous  movements  that  last  for  many 
seconds,  but  even  so,  it  is  possible  that  such  movements  are  composed 
completely  of  discrete  processes.  Since  the  perception  and  decision 
operations  that  must  constitute  the  basis  of  human  tracking  performance 
are  generally  considered  to  be  discrete  processes,  it  is  natural  to 
use  a  discrete  model  to  describe  tracking. 

The  second  point  recognizes  that  different  individuals  will 
use  very  different  processing  strategies  on  the  same  task  and  even  a 
single  individual  may  use  very  different  strategies  on  two  tracking 
tasks  that  appear  to  differ  only  slightly.  This  tends  to  frustrate 
attempts  to  model  behavior  in  tasks  like  tracking  where  the  operator 
may  choose  from  many  different  processing  strategies.  People  probably 
try  out  several  different  strategies  before  they  adopt  a  stable  strategy, 
and  even  that  final  strategy  may  undergo  frequent  modifications  in 
response  to  feedback.  But  since  the  possible  strategies  must  be  limited 
by  the  type  of  information  available  to  the  operator  and  by  the  operator's 
ability  to  interpret  that  information,  there  are  only  a  few  possibilities 
that  require  consideration. 

2.3  The  HOS  Tracking  Model 

2.3.1  Rationale 

In  our  model,  we  attempted  to  identify  the  information  that 
could  be  available  in  a  typical  tracking  task  and  the  factors  that 
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determine  how  that  information  will  be  used.  This  was  then  formulated 
into  a  model  of  how  the  information  is  processed  to  effect  control 
movements.  In  addition  to  providing  an  understanding  of  tracking  per¬ 
formance,  it  was  expected  that  this  research  would  provide  valuable 
information  for  use  in  a  general-purpose  model  of  human  performance. 

Our  approach  was  to  start  with  a  basic  model,  identify  the  deficiencies 
of  that  model,  and  then  modify  it  accordingly. 

The  model  was  developed  for  a  pursuit  tracking  task  in  which  a 
single  display  indicated  the  values  of  both  the  signal  and  the  track. 

The  value  of  the  track  was  controlled  by  the  position  of  a  rotary  knob. 
Our  basic  model  assumed  that  the  operator  repeatedly  executed  a  basic 
adjustment  cycle  consisting  of: 

(1)  Reading  the  value  of  the  signal. 

(2)  Reading  the  value  of  the  track. 

(3)  Computing  the  error  (i.e.,  the  difference  between  the 
signal  and  track  values) . 

(4a)  Terminating  the  adjustment  cycle  if  the  error  was  less  than 
a  tolerance  limit,  or 

(4b)  Computing  the  amount  of  knob  turn  required  to  make  the 

track  equal  to  the  signal  (using  the  precise  gain  factor 
that  relates  knob  movements  to  changes  in  the  track  value) , 
and  making  the  appropriate  change  if  the  error  is  greater 
than  the  tolerance  limit. 

Since  this  model  makes  no  provision  for  learning,  it  can  be  considered 
to  be  a  description  of  the  stable  performance  of  a  trained  operator. 

Even  ignoring  the  fact  that  details,  such  as  the  speed  and 
accuracy  of  the  component  processes,  have  to  be  specified,  it  is  clear 
that  inadequacies  exist  in  the  model.  For  example,  the  model  assumes 
that  the  operator  knows  the  precise  gain  factor  that  relates  changes 
in  the  knob's  value  to  changes  in  the  value  of  the  track.  While  this 
assumption  would  not  necessarily  result  in  inaccurate  performance  pre¬ 
dictions,  it  does  compromise  both  the  face  validity  and  the  generality  of 
the  model.  Moreover,  the  model  assumes  that  the  operator  does  not 
rely  on  his  experience  based  on  previous  adjustment  cycles  to  anticipate 
future  values  of  the  signal.  This  assumption  would  preclude  the  occur¬ 
rence  of  tracking  overshoots  at  signal  inflections  and  tracking  lags 
shorter  than  the  operator's  reaction  time.  Since  real  operators  do 
frequently  overshoot  the  maxima  and  minima  of  the  signal  and,  for  some 
signals,  exhibit  very  short  lags,  modifications  to  the  basic  model  were 
clearly  necessary.  The  modifications  that  were  implemented  provide  the 
model  with  the  capability  to  learn  both  a  signal  extrapolation  time 
and  an  estimate  for  the  system  gain  factor. 
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2.3.2  Notation 


In  discussing  the  modified  tracking  model,  the  following  notation 
will  be  used: 

S  =  the  operator '  s  estimate  of  the  signal  value  on  the  n*'*1 
adjustment  cycle 

S  *  =  the  operator's  extrapolated  estimate  of  the  signal 

n  value  on  the  n*1*1  cycle 

t  =  the  extrapolation  time  used  by  the  operator  on  the  n^ 
cycle 

T  =  the  operator's  estimate  of  the  track  value  on  the  n*"*1 

n  i 

cycle 

g  =  the  operator’s  estimate  of  the  gain  factor  on  the  n^ 
cycle 

A  =  the  operator's  tolerance  for  being  :'cn  target"  (assumed 
to  be  constant) 

=  the  extrapolation  time  learning  function 
=•  the  gain  factor  learning  function 
2.3.3  Sequence  of  Operations 


The  model  assumes  that  on  any  adjustment  cycle,  the  operator  per¬ 
forms  the  following  sequence  of  operations: 

(1)  Reads  the  value  of  the  signal,  S 

n 

(2)  Computes  the  new  values  for  the  extrapolation  time, 

t  =  L4-  (t  ,  ...) 

n  t  n-i 

(3)  Computes  the  extrapolated  signal  value,  S^* 

(4)  Reads  the  value  of  the  track,  T 

n 

(5a)  Terminates  the  adjustment  cycle  if  !sn*  -  T  |  £  A  ,  or 

(5b)  Computes  the  new  value  for  the  gain  factor, 

g  =  L  (g  ,...),  the  desired  knob  turn,  and  makes 
n  g  ^n-1 

the  appropriate  adjustment  if  t sn*  -  T  |  >  A. 


The  speeds  and  accuracies  for  the  display  reading  and  control 
manipulation  operations  are  automatically  assigned  by  HOS  micro-models. 
The  HOS  micro-model  for  display  reading,  described  in  detail  in  Volume 
VII  of  this  series,  models  the  reading  process  by  a  sequence  of  discrete 
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micro-absorptions  that  are  terminated  either  when  successive  estimates 
differ  by  a  small  enough  amount  that  the  operator  is  considered  to  have 
"learned"  the  value,  or  when  a  time  limit  for  the  reading  process  is 
exceeded.  The  estimated  value  of  the  display  on  each  micro-absorption 
is  obtained  by  averaging  the  actual  value  of  the  display  with  a  value 
extrapolated  linearly  from  the  previous  two  micro-absorption  estimates. 
The  time  consumed  by  each  micro-absorption  is  defined  as  a  constant 
cost  plus  an  additional  amount  dependent  on  the  dissimilarity  between 
the  estimated  values  of  the  current  and  previous  micro-absorptions . 

The  effects  of  this  display  reading  model  in  the  tracking  simulation 
studies  are: 


•  Display  values  are  always  read  to  within  a  narrowly  defined 
additive  tolerance  of  the  actual  value 

•  The  time  for  each  reading  varies  between  about  .02  seconds 
and  about  . 2  seconds 

•  The  longest  reading  times  are  associated  with  the  most 
extremely  nonlinear  changes  in  signal  values. 

The  HOS  control  manipulation  micro-model  causes  the  adjustments 
to  be  made  to  precisely  the  intended  setting.  Manipulations  require  an 
amount  of  time  that  is  a  linear  function  of  the  magnitude  of  the  adjust¬ 
ment.  The  time  consumed  in  turning  a  knob  is  given  by  the  function 

time  (in  seconds)  =  .0029A  +  .0982 
where  A  is  the  angle  in  degrees  through  which  the  knob  is  turned. 

All  operations,  other  than  display  readings  and  knob  adjustments 
(i.e.,  all  computations  and  the  one  comparison),  were  assigned  zero  time 
costs. 

2.3.4  The  Learning  Functions 


The  two  learning  functions  were  defined  so  that  the  amount  of 
information  that  the  operator  had  to  store  and  use  was  kept  to  a  minimum. 
In  order  to  be  able  to  extrapolate  the  signal  value,  it  was  necessary  to 
assume  that  the  operator  "knew"  both  the  current  and  previous  estimates 
of  the  signal  and  the  times  when  those  estimates  were  obtained.  He 
also  had  to  know  the  current  and  previous  values  of  the  track,  the  current 
working  values  for  the  gain  factor  and  extrapolation  time,  and  a  running 
average  for  the  absolute  tracking  error. 

A  gain  learning  function  must  be  sensitive  to  the  effect  of 
the  most  recent  control  manipulation,  increasing  the  estimated  gain  factor 
if  that  manipulation  was  too  large  and  decreasing  the  estimated  gain  if 
the  manipulation  was  too  small.  An  extrapolation  time  learning  function 
must  be  sensitive  to  the  extent  to  which  the  track  is  tending  to  lead  or 
lag  behind  the  signal,  increasing  the  extrapolation  time  if  the  track  is 
lagging  behind  the  signal  and  decreasing  the  extrapolation  time  if  the 
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track  is  leading  the  signal.  A  set  of  learning  functions  that  have 
these  properties  is  shown  in  Table  1.  The  parameters  <J>,  0,  the  initial 
learning  rates  for  extrapolation  and  gain,  and  6,  the  relative  size  of 
random  changes  to  the  gain,  are  input  constants.  (j>,  0,  and  6  are 
defined  on  the  range  of  0  to  1  and  are  considered  to  be  of  approximately 
the  same  magnitude. 

These  learning  functions  imply  that  the  effective  learning 
rates  on  any  adjustment  cycle  (<}>  and  0  )  are  the  product  of  the  base 
learning  rates  and  the  ratio  of  She  estimated  error  (E  )  to  the  average 
absolute  error  (A  )  on  any  cycle.  Thus,  relatively  small  changes  are 
made  to  both  the  extrapolation  and  gain  values  when  the  tracking  error 
is  relatively  small  and  relatively  large  changes  are  made  when  the 
tracking  error  is  large. 

The  extrapolation  learning  function  increments  or  decrements 
the  old  extrapolation  value  by  the  product  of  the  old  extrapolation 
value  and  the  effective  learning  rate,  incrementing  when  the  track  lags 
behind  the  signal  and  decrementing  when  the  track  leads  the  signal. 

The  gain  learning  function  determines  the  new  gain  by  taking  a 
weighted  average  of  the  old  gain  and  an  estimate  for  the  "ideal"  gain, 
the  weighting  factor  being  the  effective  learning  rate.  The  ideal  gain 
calculation  assumes  that  the  signal  will  not  change  appreciably  over 
a  single  adjustment  cycle.  Then,  if  the  actual  adjustment  on  the  n-1 
cycle  was  optimal,  the  error  on  the  cycle  would  be  zero.  The  actual 

adjustment  made  on  the  n-lst  cycle,  a,  is  given  by 


Vl  gn-l 

and  the  adjustment  that  would  have  been  optimal  for  that  cycle,  a*,  is 
given  by 


where  g*  is  the  corresponding  optimal  gain. 


Since  the  actual  adjustment  changed  the  tracking  error  from  E 
to  E  and  the  optimal  adjustment  would  have  changed  the  error  from 
to  0?  the  assumption  that  the  control  is  linear  implies  that 


“1  =  En-1 

a  E  _  -  E 
n-1  n 
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TABLE  1.  LEARNING  FUNCTIONS  FOR  PURSUIT  TRACKING 


I. 


II. 


III. 


For  running  average  of  absolute  error: 


A  =  .  9  A  +  .  1  E 
n  n-1  1  n1 


For  extrapolation: 


t  = 
n 


where  0  =  max 

n 


(1  +  0  ) 

t  _ 

if 

S  < 

s  , 

n 

n-1 

n 

n-1 

(1  -  0  ) 

t  _ 

if 

S  > 

S  , 

n 

n-1 

n 

.  n-1 

(„i„  (?  •  -2)  •  ■ 


For  gain: 

g  =  [enf  +  (i  -  e  ) ]  g  . 

n  n  n  n  n-1 

0|e 


where  0  =  min  1.9,  — 

n  I  A 


and 


2)  ,  .5)  if  B  =  0  and  < 

'  n-1  f  /  n-1 


max  mm 


f «  \  1+6  with  probability  h 


j,  +  g  with  probability  h 


if  E  =  0  or  >  1 

n-1  E  , 

n-1 


where  E  =  T  -  S  is  the  estimated  error  on  the  n  adjustment 
n  n  n  t 
cycle 
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■V 


and  that 


if 


E  -  E 
*  -  n-1  n 

g*  =  -  g 

*  E  _  yn-l 

n-1 

If,  however,  the  adjustment  causes  an  increase  in  the  error,  i.e.. 


>  1, 


"n-1 


then  this  formula  implies  that  g*  and  g  must  have  different  signs, 
corresponding  to  a  change  in  the  estimated  directional  relationship 
between  the  control  and  the  track  value.  Since  such  sign  changes  would 
result  in  a  highly  unstable  learning  process,  it  was  assumed  that  the 
operator  knows  the  directional  correspondence  between  the  control  and 
the  display.  Therefore,  the  model  permits  only  positive  estimates  for 
the  gain.  When 


>  1 


or  when 


a  new  value  for  the  ideal  gain  was  considered  to  be  indeterminable. 

Since  such  indeterminate  cases  represent  conditions  of  deteriorating 
tracking  performance,  the  new  gain  value  for  that  cycle  was  randomly 
defined  as  a  fixed  increment  or  decrement  from  the  old  gain,  each  change 
occurring  with  a  probability  of  .5. 

For  both  gain  learning  and  extrapolation  learning,  the  maximum 
change  itirde  to  either  the  gain  or  extrapolation  time  on  a  single  cycle 
is  indicated  by  the  truncation  operations  shown  in  Table  1.  These 
limitations  keep  the  gain  learning  from  being  dominated  by  the  special 
cases 


E  «  0  <  E  . 
n  n-1 


and 


E  <  0  «  E  , 
n-1  n-1 


and  extrapolation  learning  by  the  special  case; 

|e  (  »  . 

1  n 1  0 
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2.4 


Parametric  Behavior  of  the  HOS  Tracking  Model 


Despite  its  similarity  to  the  linear  learning  models  that  have 
been  extensively  studied  by  mathematical  psychologists  (Bush  and  Mosteller, 
1956;  Norman,  1968) ,  the  tracking  model  is  sufficiently  different  from 
those  models  so  that  very  different  methods  of  analysis  are  necessary. 

In  particular,  other  mathematical  models  for  learning  have  dealt  principally 
with  the  learning  of  probabilities  and  probability  distributions  rather 
than  with  the  learning  of  strategic  parameters  like  system  gain  and 
extrapolation  time.  Also,  other  learning  models  were  developed  for 
situations  in  which  learning  occurred  on  discrete  trials  for  which  the 
experimenter-controlled  events  (e.g.,  stimuli  and  rewards)  were  scheduled 
independently.  For  the  tracking  situation,  the  stimulus  is  continuous 
and  successive  learning  events  are  consequently  correlated.  Further 
complicating  the  analysis  of  the  tracking  model  is  the  fact  that  the 
learning  functions  are  discontinuous  unlike  the  learning  functions  used 
in  other  models.  And  even  if  none  of  these  features  had  precluded  an 
analysis  of  the  model,  the  fact  that  it  uses  two  interacting  learning 
functions  would  have  made  the  model  sufficiently  intractable  analytically. 
Thus,  the  only  viable  method  for  studying  the  behavior  of  the  model  is 
to  examine  the  results  of  dynamic  simulations  for  a  variety  of  different 
model  parameters  and  different  types  of  signal. 

2.4.1  Measures  of  Performance 


In  evaluating  the  behavior  of  the  tracking  model,  it  is  important 
to  consider  both  the  accuracy  of  tracking  performance  and  the  "success” 
of  the  two  learning  processes.  Two  different  tracking  performance 
measures,  root-mean- squared-error  (RMSE)  and  percent-time-on-target 
(PTOT)  were  used  to  assess  tracking  accuracy,  since  it  would  have  been 
impossible  to  predict  which  measure  would  be  most  sensitive  to  the  per¬ 
formance  differences.*  The  success  of  the  learning  process  was  evaluated 
based  on  its  stability  (i.e.,  variance  and  regression  over  time)  and 
central  tendency. 

Each  tracking  trial  lasted  for  60  seconds  of  simulated  time  and 
started  with  an  initial  gain  value  of  200  display  units  per  degree  of 
knob  turn  (the  actual  gain  of  the  simulated  system)  and  an  initial 
extrapolation  time  of  .1  second  (somewhat  less  than  the  adjustment  cycle 
time  of  the  simulated  operator).  Performance  data  were  collected  only 
for  the  last  50  seconds  of  each  trial.  Trials  were  conducted  for  several 
values  of  the  model  parameters  0,  6,  and  0,  the  objective  being  to  get 
a  general  idea  of  how  the  parameters  governed  model  performance.  In 
all  cases,  the  signal  was  sinusoidal  with  an  offset  of  10,000  units  and 
an  amplitude  of  either  4000  or  8000  units.  For  each  combination  of 
model  parameters,  the  first  tracking  trial  was  conducted  with  a  signal 
frequency  of  .1  cycles/second  and  subsequent  trials  with  the  same 


*  For  present  purposes,  PTOT  is  arbitrarily  defined  as  the  percentage 
of  time  that  the  track  is  within  400  units  of  the  signal. 
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parameters  used  signal  frequencies  increased  by  a  factor  of  until 
performance  was  degraded  to  an  uninteresting  level. 

2.4.2  Simulation  Studies 

The  results  of  these  parametric  runs  are  shown  in  Table  2. 

Section  A  of  the  table  demonstrates  the  baseline  performance  —  without 
learning,  but  with  a  precisely  correct  gain  value  and  a  fairly  small 
extrapolation  time.  Note  that  the  two  overall  performance  measures, 

RMSE  and  PTOT,  appear  to  be  about  equally  sensitive  to  each  increment 
in  signal  frequency.  In  Section  B  of  Table  2,  only  the  gain  learning 
features  of  the  model  were  being  exercised  and,  in  Section  C,  only  the 
extrapolation  time  learning  component  was  active.  Sections  D  and  E 
indicate  how  the  model  behaves  when  both  the  gain  and  extrapolation  time 
learning  features  are  exercised  together.  In  Section  E,  the  signal  had 
twice  the  amplitude  of  the  signal  used  in  Section  D.  Finally,  Section 
F  displays  model  performance  when  both  gain  and  extrapolation  time  are 
learned  but  at  rates  that  are  twice  as  large  as  those  used  in  other  cases. 
Data  for  only  three  different  signal  frequency  values  are  presented  in 
Section  F  because  during  the  simulation  with  the  fourth  frequency  (.2828 
cycles  per  second) ,  an  unrealistically  large  estimate  for  the  extrapola¬ 
tion  time  resulted  in  a  knob-turn  that  was  not  completed  during  the  span 
of  the  simulation. 

Figure  1  shows  how  the  learned  gain  value  varies  with  signal 
frequency.  From  Tabl ..  2  it  can  be  seen  that  the  points  with  the  largest 
gain  values  correspond  to  trials  with  large  regression  coefficients,  so 
the  results  are,  in  fact,  underestimates  of  any  ultimately  stable  gain 
value  that  might  (or  might  not)  be  attained.  When  only  gain  learning 
is  present  (case  b) ,  the  learned  gain  is  fairly  accurate  (with  respect 
to  actual  system  gain)  at  the  lowest  frequency.  It  stabilizes  at  a  value 
about  35%  above  the  actual  system  gain  at  the  midrange  of  frequencies 
from  .2  to  .4  cycles/second  and  becomes  unstable  above  .5656  cycles/ 
second.  When  both  gain  and  extrapolation  time  are  being  learned  (case  d) , 
similar  stable  gain  values  are  learned  for  low  signal  frequencies,  but 
the  model  appears  to  become  unstable  at  a  lower  frequency  (by  .4  cycles/ 
second)  than  in  case  b.  Curiously,  when  the  signal  amplitude  is  doubled 
and  both  learning  processes  are  operative  (case  e) ,  a  fairly  high  gain 
value  is  learned  at  the  lowest  signal  frequency,  while  lower,  more 
accurate  values  are  learned  for  slightly  higher  frequencies,  with 
instability  occurring  at  roughly  the  same  point  as  at  the  smaller  signal 
amplitude.  When  the  learning  rates  for  both  processes  are  doubled 
(case  f) ,  the  learned  gain  appears  to  be  virtually  the  same  as  when 
the  smaller  learning  rates  are  used. 

Figure  2  shows  how  the  learned  extrapolation  time  varies  with 
signal  frequency.  Again,  the  regression  coefficients  in  Table  2,  indicate 
that  the  larger  extrapolation  times  correspond  to  the  means  of  values 
that  increase  systematically  with  simulation  time.  All  model  cases 
show  basically  the  same  relationship  between  learned  extrapolation 
time  and  signal  frequency  when  the  signal  amplitude  is  constant.  At 
the  lowest  frequency  (.1  cycles/second)  a  stable  extrapolation  time  of 


r 


TABLE  2.  RESULTS  OF  PARAMETRIC  RUNS 


SIGNAL  SIGNAL 
AMP 


STAND  REGRESS. 
DEV 


EXTRAPOLATION 

TIME 

STAND  REGRESS. 

COE  F.  RMSE 


265.0  |  88.5 


71.2 


425 


1264.9  |  22.4 


175.56  10.00  -.041 


15.66 


24.40 


16.84 


265.83  9.72 


320.27  54.53  |  2.886 


642.60  200.61  11.000 


385  3  62.3 


782.9  28.5 


1425.3  I  14.5 


.000 

169.2 

99.6 

.001 

322.0 

83  5 

.000 

786.3 

45.2 

.001 

1 689. 1 

14.6 

.213 

45125  0 

1.0 

■a 

■89 

1.9 

.1000 

202.95 

.1414 

231.24 

.2000 

359.49 

.2828 

246.57 

.4000 

535.29 

i 

I 

44.58  1.998 


30.74 


301.4  84.9 


14121  3  1.4 


352.34  56.67 


244.46  35.14 


194.00  42.62  -2.162 


195.72  24.66 


215.93 


771.80  583.47 


4.133  3.814 


3.533 


27039.4  i  1.0 


201.75'  96.0 


2.600 

.400 

.1 

52 

IBS 

2 

39.0 

LEARNED  EXTRAPOLATION  TIME  (SECS.) 
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about  .22  seconds  is  learned.  At  higher  signal  frequencies,  larger 
extrapolation  times  are  obtained,  the  relationship  being  approximately 
exponential  over  the  midrange  of  frequencies.  The  extrapolation  time  peaks 
at  .4  cycles  per  second  in  all  cases  and  is  followed  by  a  slightly  lower, 
but  still  unstable,  value  at  the  next  higher  frequency.  For  the  larger 
signal  amplitude  (case  e) ,  the  major  difference  occurs  at  the  lower  signal 
frequencies  where  the  extrapolation  time  obtained  is  considerably  larger 
than  for  the  other  case. 

In  Figure  3,  the  overall  tracking  performance  accuracy  measured 
by  RMSE  is  plotted  as  a  function  of  signal  frequency.  In  Figure  4,  accuracy 
as  measured  by  PTOT  is  plotted.  Both  plots  show  the  same  general  trends, 
though  the  trends  are  somewhat  more  apparent  in  Figure  3  because  of  the 
scales  used.  Approximately  the  same  performance  is  achieved  when  neither 
quantity  is  learned  and  when  only  the  gain  is  learned.  When  the  extrapola¬ 
tion  time  is  learned,  performance  is  essentially  unaffected  by  the  addition 
of  gain  learning.  Whenever  the  extrapolation  time  is  learned,  performance 
is  better  than  the  corresponding  baseline  level  (case  a)  at  the  two  lowest 
frequencies  and  worse  than  baseline  at  the  higher  frequencies.  Tracking 
accuracy  is  poorest  at  .4  cycles  per  second,  the  point  at  which  the  extra¬ 
polation  time  learning  process  was  least  stable.  Not  surprisingly,  per¬ 
formance  with  the  doubled  signal  amplitude  is  uniformly  worse  than  with 
the  standard  amplitude  signal  except  at  the  highest  frequencies  where  per¬ 
formance  was  comparable  to  that  obtained  in  the  other  extrapolation  time 
learning  situations. 

2.4.3  Conclusions 

These  results  indicate  that  the  model  performance  is  reasonable  for 
most  situations.  However,  some  unexpected,  but  unmistakable,  trends  are 
evident.  The  fact  that  the  learned  model  parameters  do  not  stabilize  in 
some  situations  is  disturbing,  since  it  is  difficult  to  conceive  of  a  cor¬ 
responding  feature  in  human  tracking  performance.  However,  large  estimated 
gain  values  correspond  to  small  knob  adjustments.  Thus,  the  "unstable" 
gain  values  obtained  at  high  frequencies  tend  to  dampen  the  knob  turning 
responses,  which  would  have  been  more  erratic  without  gain  learning.  When 
the  signal  frequency  is  slow  enough  to  permit  fairly  successful  tracking, 
the  gain  learning  model  behaves  quite  reasonably,  converging  near  the 
actual  system  gain. 

The  extrapolation  time  learning  exhibits  some  suspicious  charac¬ 
teristics  with  unstable  learning  at  frequencies  above  . 2828  cycles  per 
second  and  with  larger  rather  than  smaller  times  being  obtained  as  the 
frequencies  increase.  Since,  however,  it  is  not  clear  how  people  extra¬ 
polate  the  signal  in  such  a  situation,  the  trend  may,  in  fact,  constitute  a 
valid  description  of  a  real  component  of  human  tracking  behavior. 


Although  detailed  time  plots  of  signal  and  crack  have  not  been 
included  because  of  their  low  information  content,  it  should  be  noted 
that  extrapolation  learning,  either  alone  or  with  gain  learning,  produced 
overshoots  at  signal  inflections  and  definite  compensation  for  the 
operator’s  tendency  to  lag  behind  the  signal.  Gain  learning  alone  did 


ROOT  MEAN  SQUARED  ERROR 


not.  Gain  learning  could  have  been  expected  to  generate  such  features  j 

if  the  learned  gain  value  were  smaller  than  the  actual  system  gain,  \ 

but  that  did  not  happen  reliably  in  any  of  the  situations  that  were  ] 

simulated.  j 

} 

2.4.4  Recommendations  \ 

Before  further  development  of  our  model  is  undertaken,  a 
systematic  comparison  of  the  model  predictions  with  the  performance 
of  real  human  operators  should  be  made.  The  experiments  that  we  have 
simulated  should  be  a  good  starting  place  for  such  a  study.  In  addition, 
it  would  be  desirable  to  compare  both  human  and  model  performance  with 
signals  more  complex  than  a  simple  sine  wave.  In  this  context,  it  would 
be  valuable  to  study  the  extent  to  which  the  model  could  be  characterized 
as  a  linear  system.  Such  investigations  will  probably  result  in  modifica¬ 
tions  to  the  model,  especially  to  the  extrapolation  learning  component, 
but  without  such  studies,  it  is  impossible  to  determine  what  changes 
are  necessary. 


3. 


STUDIES  OF  WORKLOAD 


Tasks  that  are  vulnerable  to  degradation  by  interruption  could 
reasonably  be  said  to  impose  a  greater  workload  on  the  human  operator, 
therefore,  if  the  workload  requirement  of  a  continuous  task  like  track¬ 
ing  could  be  characterized  by  a  performance  function  that  related  per¬ 
formance  when  the  primary  task  is  interrupted  for  any  given  percentage 
of  time  to  uninterrupted  tracking  performance,  a  major  step  toward  defin¬ 
ing  a  general  workload  metric  would  have  been  achieved.  If,  however, 
the  performance  function  was  dependent  on  the  particular  interruption 
schedule,  even  though  the  total  amount  of  interruption  time  was  fixed, 
then  it  would  be  much  less  useful  in  assessing  workload.  The  studies 
to  be  described  in  this  section  were  undertaken  to  determine  the  feasi¬ 
bility  of  a  general  workload  metric  using  HOS's  ability  to  simulate 
interrupted  tracking  tasks.  While  the  validity  of  these  studies  are, 
of  course,  dependent  upon  a  validation  of  the  HOS  tracking  model,  the 
studies  do  provide  interesting  insight  into  the  workload  metric  issue 
and  a  solid  set  of  performance  data  for  eventual  comparison  with  experi¬ 
mental  data. 

Since  there  is  no  a  priori  way  to  determine  which  might  be  most 
sensitive  to  the  factors  being  examined,  both  root-mean-squared  error 
(RMSE)  and  percent  time-on-target  (PTOT)  were  used  as  measures  of  per¬ 
formance.  For  PTOT,  five  different  criteria  of  “on-target"  were  used  to 
insure  that  any  real  effects  would  be  detected. 

The  specific  code  that  implements  the  tracking  model  and  other 
input  data  required  by  these  studies  is  listed  in  Appendix  II.*  Some 
minor  modifications  were  made  to  HOS  in  order  to  simplify  the  required  data 
processing.  These  modifications  are  discussed  in  Appendix  III.  In  order 
to  compute  the  performance  measures  reported  in  these  studies,  a  special 
purpose  data  analysis  program  (called  EVAL)  was  written.  That  program  is 
described  and  listed  in  Appendix  IV.  Each  tracking  trial  in  the  four 
studies  to  be  described  lasted  for  220  simulated  seconds.  Performance 
data  were  collected  only  over  the  last  200  seconds  in  order  to  avoid 
transient  effects.  The  parameters  used  for  all  of  the  workload  studies  to 
be  reported  are: 


9  =  .05 
5  =  .02 
<t>  =  .05 

and,  except  for  Study  4, 

A  =  800  units. 


*  The  code  and  data  used  for  the  parametric  studies  described  in 
Section  2  are  listed  in  Appendix  I. 


3.1 


Study  1  —  The  Effects  of  Task  Difficulty  and  Time-Sharing  on 
Tracking  Performance 


This  study  examined  the  sensitivity  of  the  HOS  tracking  model 
to  changes  in  signal  characteristics  and  secondary  task  loading.  It 
was  expected  that  as  the  frequencies  and  amplitudes  of  the  component 
sine  waves  that  form  the  signal  became  faster  and  larger,  respectively, 
performance  should  degrade.  Similarly,  it  was  expected  that  performance 
would  be  worse  with  secondary  task  interruptions  that  without  them. 

These  hypotheses  were  tested  by  requiring  the  simulated  operator  to 
track  three  different  signals  both  with  and  without  the  secondary  task 
for  a  total  of  six  simulation  trials.  The  three  signals  were  the  com¬ 
posite  of  three  sine  waves  of  different  frequencies  and  amplitudes,  as 
shown  below: 


AMP 

FREQ  (H2) 

AMP 

FREQ  (Hz) 

AMP 

FREQ  (Hz) 

Signal 

1 

2000 

1/10 

1400 

1/7 

800 

1/4 

Signal 

2 

4000 

1/20 

2800 

1/14 

1600 

1/8 

Signal 

3 

4000 

1/10 

2800 

1/7 

1600 

1/4 

Thus,  all  three  signals  had  the  same  basic  shape.  Signal  1  differed 
from  Signal  3  only  by  having  the  component  amplitudes  reduced  by  a  factor 
of  two  and  Signal  2  differed  from  Signal  3  by  having  the  component  fre¬ 
quencies  reduced  by  a  factor  of  two. 

The  results  for  Study  1  are  presented  in  Table  3.  They  indicate 
definite  performance  degradation  when  either  the  frequency  or  the  ampli¬ 
tude  of  the  signal  increases  or  when  secondary  task  interruptions  occur. 
Nothing  further  can  be  concluded  from  these  results  about  the  relation¬ 
ship  between  performance  and  signal  characteristics  since  only  two  sets 
of  frequencies  and  amplitudes  were  used.* 

The  data  in  Table  3  clearly  indicate  that  when  examining  the 
performance  decrements  caused  by  interruptions  to  the  primary  task,  one 
must  be  aware  of  the  implications  of  using  a  specific  performance  measure. 
For  example,  if  one  uses  the  differences  between  displayed  scores  as  the 
measure  of  performance  decrement,  then,  based  on  RMSE,  performance  on 
Signal  1  is  least  degraded  by  introducing  interruptions  and  performance 
on  Signal  3  is  most  degraded.  If  differences  between  PTOT  are  taken, 
though,  exactly* the  opposite  results  are  obtained.  If  the  ratios  of 
scores  rather  than  the  differences  are  used,  yet  another  ordering  is 
obtained  and  if  the  PTOT  scores  are  converted  to  "time  off  tavget" 
scores  still  another  ordering.  One  is  forced  to  conclude,  therefore, 


*  However,  the  studies  described  in  Section  2  examined  the  relationship 
between  model  performance  and  signal  characteristics  without  any 
secondary  task  interruptions  in  more  detail. 


> 

Q 

Z) 


c/) 

LL 

o 

00 

h- 


D 

00 

LU 

OC 


CO 


m 

< 

h- 


co 


O 
OC  oo 
K-  ^ 
2  “) 
o  § 

O 


o 

o 

CO 


o 

o 

LO 


o 

o 

CO 


OC 
C/3  o 
2  OC 
OC  OC 
LU 


< 

1 1 
H  I 

2  -J 


2 

O  Oj 

<  < 

2  od 

OC  D 

UJ  Q 


-  £° 

<  O 


*  2 


to  £E 
<  LU 

a. 


O 

< 

OC 

H 


o 


55  LU 

s? 

O  u_ 

H  o 


2  O 

2  2 

co 


CO  CO 

cy>  r* 

CO  «— 


o  o 
CO  o’ 


o  o 

O  CO 


LO  LO 
t“  LO 
CM  «- 


o  o 
o  o 

CO  00 


o  o 
o  o 
CO  CO 


o  o 


o  o 


o  o 

CD  CO 


o  o 
to  o 


o  o 
o*  o 

O  CO 


i-  r-  CM  CM 


LO  G) 
CM  O 
CO  CO 


LO 

1^* 

CO 

r- 

03 

CO  co 

T— 

00 

1- 

f— 

o> 

00  CO 

CO 

03 

9 

CO* 

CM 

lo* 

CO  CM 

CM 

CO 

CO 

CM 

h» 

CO  CM 

** 

r** 

d  d 

o 

r** 

LO 

LO 

CO 

r* 

r- 

o 

LO 

CM 

CO 

LO 

CM  03* 

r- 

r~ 

03 

CO 

CO  LO 

LO 

CO 

03  LO 

f*. 

in 

03 

T— 

03  «-* 

f— 

CO 

O) 

CO 

03  r- 

03 

CO 

LO 

r* 

CM  LO 

O 

LO 

r“ 

CD  CM 

r- 

LO 

** 

LO  CO 

03 

CO 

t— 

CM 

CO 

o  o 
o  o 

CO  CO 


o  o 


o  o 

CO  to 


o  o 

CO  CO 


o  o 
to  d 


o  o 
o  o 

O  CO 


that  any  scaling  of  task  difficulty  according  to  the  size  of  the  per¬ 
formance  decrement  produced  by  introduction  of  a  secondary  task  is 
arbitrary  and  must  be  used  with  caution,  if  at  all. 

3.2  Study  2  —  The  Effect  of  Length  of  "Alternation  Period"  on 

Primary  Task  Performance 


There  are  many  reasons  why  one  would  expect  tracking  performance 
to  vary  when  the  percentage  of  interruption  time  is  held  constant  and 
the  specific  time  for  each  primary  and  secondary  task  execution  is 
varied.  For  one  thing,  as  the  length  of  the  interruption  increases, 
the  farther  (on  the  average)  the  signal  is  likely  to  have  drifted  from 
the  last  "acquired"  position.  However,  increasing  primary  task  execu¬ 
tion  time  gives  the  operator  more  time  to  make  corrections  once  he 
returns  to  the  primary  task.  Therefore,  it  is  more  likely  that  he 
will  be  able  to  resume  satisfactory  tracking  of  the  signal.  Returning 
to  stable  performance  after  an  interruption  also  requires  that  the  two 
component  learning  processes  and  error  compensation  have  time  to 
stabilize.  Thus,  the  recovery  time  is  likely  to  be  dependent  on  the 
learning  rates.  Therefore,  varying  the  alternation  period  between 
tasks  could  lead  to  either  improved,  unchanged  or  degraded  performance 
depending  on  the  choice  of  signal  characteristics  and  model  parameters. 

Study  2  addresses  these  questions.  A  single  signal  was  used 
for  all  trials  (Signal  3  from  Study  1)  and  six  different  alternation 
periods  were  examined.  In  each  case,  sixty  percent  of  the  time  was 
spent  on  the  primary  task  and  forty  percent  on  the  secondary  task. 

The  length  of  the  primary-secondary  alternation  cycle  was  varied  from 
2.5  seconds  to  15  seconds  in  2.5  second  increments. 

The  results  of  this  study  are  shown  in  Table  4  and  plotted  in 
Figure  5.  Both  RMSE  and  PTOT  are  negatively  accelerated  function;. 

All  PTOT  scores  indicate  that  the  best  performance  for  the  particular 
combination  of  model  and  signal  parameters  used  was  obtained  at  an 
alternation  period  of  10  seconds,  although  the  drop-off  at  longer 
periods  is  very  slight.  It  is  not  clear  whether  RMSE  would  continue 
to  rise  at  alternation  periods  longer  than  those  examined  but  it 
definitely  increases  over  the  lower  part  of  its  range  and  is  near  its 
maximum  value  at  10  seconds.  Thus,  the  alternation  period  which  PTOT 
indicates  as  optimal  is  clearly  sub-optimal  by  the  RMSE  measure. 

Tracking  performance  degrades  at  the  beginning  of  an  interrup¬ 
tion  and  recovers  after  an  interruption.  Figure  6  shows  the  degradation 
and  recovery  time  trajectories  as  measured  by  PTOT  with  a  criterion 
tolerance  of  1600  units.  The  envelopes  marked  "recovery"  and 
"deterioration"  are  the  envelopes  of  the  averaged  second-by-second 
scores  for  the  six  trials  in  the  study.  The  dashed  curves  are  "eye¬ 
ball"  fits  based  on  the  two  recovery  points  (i.e.,  the  average  of  the 
first  second  and  the  average  of  the  second  second)  plotted  for  each 
alternation  period.  The  trajectories  indicate  that  after  two  seconds 
of  time  on  the  primary  task,  recovery  is  virtually  complete.  Recovery 
tends  to  be  slower  at  the  ten-second  alternation  period  than  at  other 
periods,  raising  further  doubts  about  the  PTOT  maximum  noted  above. 
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i  Study  3  —  The  Effect  of  Interruption  Time  on  Tracking  Performance 

If  the  time  spent  working  on  the  primary  task  after  each  inter¬ 
ruption  is  fixed  and  only  the  duration  of  interruptions  is  allowed  to 
vary,  then  longer  interruptions  should  result  in  poorer  overall  tracking 
performance.  Study  3  examines  this  effect.  Signal  3  of  Study  1  was 
again  used  for  all  trials.  For  each  trial,  the  simulated  operator 
was  permitted  to  work  for  six  seconds  on  the  primary  task  without 
interruption.  The  length  of  the  interruptions  were  varied  from  zero 
to  six  seconds  in  one  second  increments  so  that  the  percentage  of  time 
spent  on  the  primary  task  varied  from  100  to  50  percent. 

The  results  for  this  study  are  listed  in  Table  5  and  plotted 
in  Figure  7.  The  lines  in  Figure  7  are  least-square  fits.  As  is  clear, 
both  RMSE  and  PTOT  vary  linearly  with  the  percentage  of  time  on  the 
primary  task.  It  is  interesting  to  note  that  the  y-intercepts  of  the 
PTOT  fits  are  approximately  the  time-on-target  frequencies  that  would 
be  expected  if  the  operator  spent  no  time  on  the  tracking  task  and  if 
the  signal  values  were  uniformly  distributed  over  the  signal  range 
(1600  units  to  18,400  units)  while  the  track  element  was  fixed  at  some 
value  within  that  range.* 

3.4  Study  4  —  The  Effect  on  Tracking  Performance  of  the  Operator 's 

Criterion  of  Acceptable  Performance 


The  effort  that  an  operator  will  expend  in  performing  a  task  is 
clearly  dependent  on  what  he  considers  acceptable  performance  to  be. 

If  the  task  is  such  that  the  operator  has  difficulty  achieving  accept¬ 
able  performance,  then  we  would  expect  him  to  devote  all  of  his  avail¬ 
able  effort  to  the  task  or  else  reassess  his  objectives.  If  acceptable 
performance  can  be  achieved  without  devoting  full  effort  to  the  task, 
then  some  of  his  reserve  capacity  might  be  available  for  another  task. 
Therefore,  if  we  were  to  give  an  operator  a  series  of  tracking  tasks, 
each  with  a  narrower  criterion  of  acceptable  performance,  performance 
should  improve  until  the  criterion  became  smaller  than  the  attainable 
performance.  When  too  narrow  a  criterion  was  used,  performance  might 
degrade  simply  because  the  narrow  criterion  kept  the  operator  from 
tracking  the  lower  frequency.  Such  experiments  would  be  virtually 
impossible  to  perform  with  a  real  human  operator,  since  there  is  no 
reliable  way  to  determine  or  control  the  internal  criterion  the  operator 
is  using.  With  HOS,  however,  the  criterion  of  acceptable  performance  can 
be  directly  manipulated  and  its  effects  on  tracking  performance  observed. 

Study  4  examines  the  effects  of  varying  the  operator's  internal 
criterion  of  successful  performance  for  two  different  interruption 
cycles.  The  operator’s  criterion  was  varied  from  0  to  800  in  200  unit 
increments.  The  two  interruption  cycles  were  six  seconds  of  tracking 


*  The  expected  PTOT  values  are  19.0  percent  for  a  criterion  of  1600, 
9.5  percent  for  a  criterion  of  800,  6.0  percent  for  a  criterion  of 
500,  3.6  percent  for  a  criterion  of  300,  and  1.2  percent  for  a 
criterion  of  100. 
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followed  by  a  two  second  interruption  and  six  seconds  of  tracking 
followed  by  a  four  second  interruption.  Signal  3  from  Study  1  was  again 
used  for  all  trials. 


The  results  of  this  study  are  shown  in  Table  6.  Every  decrease 
in  the  criterion  produces  an  increase  in  the  number  of  control  adjust¬ 
ments  but  the  performance  measures  remain  essentially  the  same.  Neither 
RMSE  nor  PTOT  for  the  two  largest  target  sizes  show  any  systematic 
changes.  However,  PTOT  for  the  three  smallest  target  sizes  (i.e., 

500,  300,  and  100  units)  do  show  an  improvement  in  tracking  performance 
with  each  narrowing  of  the  criterion  and  interruption  cycle  time. 

The  findings  of  this  study  are  interesting  in  that  they  indicate 
how  the  apparent  workload  imposed  by  a  tracking  task  can  be  changed 
when  the  operator  adjusts  his  internal  criterion  of  acceptable  perfor¬ 
mance.  When  there  are  no  secondary  task  interruptions,  the  operator 
may  achieve  acceptable  performance  with  a  relatively  loose  criterion 
tolerance.  When  the  tracking  task  is  periodically  interrupted  by  the 
demands  of  other  tasks,  however,  the  operator  may  have  to  narrow  his 
criterion  in  order  to  maintain  performance.  Of  course,  if  it  was  the 
operator's  responsibility  to  schedule  the  interruption  sequence,  he 
might  relax  his  criterion  of  acceptable  performance  in  order  to  minimize 
adjustment  time.  He  could  then  work  on  the  other  tasks,  thereby  keeping 
each  tracking  interruption  brief.  Therefore,  while  it  is  apparent 
that  the  operator's  workload  increases  whenever  his  criterion  is  narrowed 
(other  things  being  equal) ,  these  studies  indicate  that  measures  like 
RMSE  and  PTOT  might  not  show  any  change  in  performance. 


TABLE  6.  RESULTS  OF  STUDY  4 


< 


~  to 

o  »- 

OC  CO 

t-  D 
21 

og 

o  < 


o 

o 


o 

o 

ro 


o 

o 

CO 


o 

o 

CO 


OC 
c/j  O 
S  OC 

cc  cc 

U4 


CM 

CM 

r- 

i— 

co 

O 

CO 

CO 

CO 

05 

CO 

CO 

5 

*— 

in 

o> 

in 

05 

CO 

8 

55  c? 

3 

CO 

in 

O 

O 

«— 

o 

05 

_ 

in 

CM 

CO 

IN 

d 

o 

CM 

CM 

CO 

IN 

o> 

d 

r- 

r- 

f— 

*— 

T- 

** 

03 

q 

o 

a> 

o 

M 

q 

q 

o> 

id 

o> 

d 

CM 

8 

cd 

CM 

id 

IN 

o 

CM 

CM 

co 

CO 

CM 

CM 

CM 

CM 

co 

05 

CO 

t— 

O) 

co 

IN 

CM 

q 

IN 

q 

d 

id 

3 

IN 

o' 

IN 

id 

o 

05 

CM 

•ct 

** 

Kf 

CO 

co 

O’ 

CO 

CO 

co 

CM 

r- 

CM 

o 

CO 

00* 

CM* 

8 

csi 

<r- 

r- 

d 

CM* 

in 

CO 

ID 

co 

w 

in 

in 

in 

co 

CO 

w 

IN 

to 

in 

CM 

in 

in 

v 

IN 

cd 

cd 

O) 

cd 

cd 

3 

cd 

cd 

IN 

IN 

IN 

fN 

fN 

IN 

co 

co 

ID 

to 

s 

*»• 

CO 

g> 

co 

IN 

CO 

»— 

co 

i 

CM 

Q> 

CM 

in 

CM 

CM 

T*~* 

05 

CM 

r- 

CM 

*■* 

o 

r>« 

8 

O 

CM 

CO 

r 

CO 

8 

< 

SP 

PS 

2  -J 


o  o  o  o  o 

o  <5  o  o 

00  CO  CM 


©  o  o  o  o 

o  o  o  o 

co  co  ^  cm 


2 

o 

p 

< 

2 

OC 

UJ 


to 

2 


> 

2 

5 

o 

o 

o 

o 

O 

UI 

CM 

CM 

CM* 

CM 

H 

Q 

< 

cc 

u 

a 

< 

O 

o 

© 

o 

C/5 

OC 

cd 

CO 

CO 

CO 

o 

H 

o 

cc 

Ui 

a 

2 

o 

o 

cd 

8.0 

o 

cd 

8.0 

H 

* 

Ui 

o 

-J 

< 

5 

© 

o 

o 

o 

p 

P 

fS 

in 

id 

id 

2 

o 

U. 

IN 

IN 

rN 

5 

H 

o 

o 

< 

-J 

cc 

< 

2 

o 

a 

2 

co 

co 

co 

co 

in  fN 


o  o  o  o  o 
^ 


©  o  o  o  o 

cd  cd  to  to  cd 


©  ©  o 

odd 


o  o 
d  o 


o  o  o  o  o 

d  d  d  o  d 

co  to  to  to  to 


co  n  co  co  to 


30 


4. 


CONCLUSION 


The  parametric  studies  in  Section  2  and  the  workload  studies  in 
Section  3  demonstrate  that  the  HOS  tracking  model  exhibits  the  types 
of  performance  features  that  would  be  expected  of  a  human  operator. 

Some  of  its  features  were  unexpected  but  no  totally  implausible  findings 
were  obtained.  The  instability  of  the  learning  process  for  some  param¬ 
eter  values  and  for  some  signal  characteristics  is  disturbing  and 
deserves  further  investigation.  The  results  obtained  in  these  studies 
deserve  to  be  replicated  with  real  human  operators  in  order  both  to 
validate  the  model  and  to  corroborate  our  findings.  Whether  or  not  such 
research  indicated  that  modifications  to  the  tracking  model  were  neces¬ 
sary,  the  potential  usefulness  of  HOS  in  the  assessment  of  the  workload 
characteristics  of  crew  station  designs  has  been  clearly  demonstrated. 
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APPENDIX  I 

HOPROC  CODE  AND  HOS  INPUTS  FOR  PARAMETRIC  STUDIES 


This  appendix  contains  a  listing  of  the  HOPROC  code  and  HOS  inputs 
used  in  the  parametric  studies  presented  in  Section  2.  A  full  discussion 
of  HOPROC  and  HOS  data  card  formats  is  presented  in  Volume  II  of  this 
series,  the  HOS  Users'  Guide. 
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APPENDIX  II 

HOP ROC  CODE  AND  HOS  INPUTS  FOR  WORKLOAD  STUDIES 


This  appendix  contains  a  full  listing  of  the  HOPROC  code  and  HOS 
inputs  used  in  the  workload  studies  described  in  Section  3  of  this 
report.  A  full  discussion  of  HOPROC  and  HOS  data  card  fromats  is  pre¬ 
sented  in  Volume  II  of  this  series,  the  HOS  Users'  Guide. 
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CHANGES  TO  HOS  FOR  TRACKING  AND  WORKLOAD  STUDIES 


The  interrupted  tracking  simulations  described  in  Section  3 
required  several  minor  modifications  to  HOS  *  These  changes  replaced  the 
normal  operator-controlled  sequencing  of  procedures  by  a  time-controlled 
sequence  dependent  upon  an  input  alternation  period.  The  modified  version 
of  HOS  was  also  used  for  the  parametric  studies  reported  in  Section  2 
for  which  the  duration  of  each  interrupt  was  set  to  zero.  The  changes 
made  to  the  standard  version  of  HOS  were; 

(1)  Four  new  variables  --  TPRIM,  TBETW,  DURINT,  and  INDPRO  were 
added  to  COMMON  JUNK.  TPRIM  is  the  time  when  each  execution 
of  the  primary  procedure  begins.  TBETW  is  the  time  between 
interrupts  (i.e.,  the  time  spent  on  the  primary  task  for 
each  alternation  period) .  DURINT  is  the  duration  of  each 
interrupt.  INDPRO  is  the  indicator  of  which  procedure  should 
be  executed  at  any  time  (INDPRO  »  1  for  the  primary  procedure 
and  INDPRO  =  2  for  the  secondary  procedure) . 

(2)  Four  cards 

IF  (INDPRO. EQ.2.0R.STIME.LT. (TPRIM+TBETW) )  GO  TO  1025 


INDPRO  2 
GO  TO  1040 
1025  CONTINUE 

were  inserted  immediately  after  statement  label  1020  in  the 
HOS  main  routine. 

(3)  Three  cards 

IF (IP. NE. 28)  GO  TO  280 
TPRIM  =  STIME 
INDPRO  =  1 

were  inserted  immediately  prior  to  statement  label  280  in 
subroutime  ENDPROC. 

(4)  One  card 

IF (MARGIN (1) .EQ„ 10HINTERRUPTI)  GO  TO  820 

was  inserted  in  the  list  of  IF  statements  between  statement 
lables  301  and  310  in  the  subroutine  INPROC. 
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(5)  Six  cards 


820  CONTINUE 

CALL  NEWORD 
DURINT  =  DIGITS 
CALL  NEWORD 
TBETW  =  DIGITS 
GO  TO  290 

were  inserted  immediately  prior  to  the  END  card  in  the  sub¬ 
routine  INPROC. 

(6)  One  card 

IF ( IC . NE . 1 )  IC=INDPRO+l 

These  changes  require  one  additional  input  data  card  that  is  not 
required  by  the  standard  version  of  HOS.  That  card  has  the  words 
INTERRUPTION  DATA  starting  in  column  1  and  two  numbers  starting  in  column 
21.  The  first  number  specifies  the  length  of  each  interruption  and  the 
second  number  specifies  the  length  of  each  primary  task  execution. 
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APPENDIX  IV 

DESCRIPTION  AND  LIST  OF  EVAL 


The  analysis  program  EVAL  computes  a  variety  of  tracking  perfor¬ 
mance  statistics  from  data  appearing  on  one  of  the  output  files  created 
by  HOS.  This  file,  originally  created  for  use  by  the  HODAC  graph-generat¬ 
ing  subroutine  TGRAPH,  is  a  series  of  two-word  (60  bits  per  word)  micro¬ 
reports  that  describe  simulation  events.  The  low-order  twelve  bits  of 
the  first  word  contains  a  message  number  that  identifies  the  report. 

The  only  message  numbers  recognized  by  EVAL  are: 

NUMBER  MESSAGE 

13  START  TO  COMPUTE  . . . 

35  BEGIN  MANIPULATING  ...  WITH  LEFT  HAND 

36  COMPLETE  MANIPULATING  . . .  WITH  LEFT  HAND 

62  COMPLETE  COMPUTATION  OF  . . . 

80  ACTUAL  VALUE  OF  . . .  CHANGED 

Other  messages  appearing  on  the  file  are  ignored  by  EVAL. 

The  next  twelve  bits  of  the  first  word  contain  the  dictionary 
entry  number  of  the  device,  function,  or  procedure  referenced  by  the 
message.  The  highest  order  30  bits  contain  the  simulation  time  at  which 
the  message  was  produced.  For  messages  13,  35,  36,  and  62,  the  second 
word  contains  the  estimated  value  (EST)  of  the  referenced  item  in  its 
low  order  50  bits  and  the  integerized  value  of  the  hab  strength  of  the 
item  multiplied  by  1,000  in  its  high  order  10  bits.  For  message  number 
80,  the  full  60  bits  of  the  second  word  are  used  to  represent  the  new 
actual  value  of  the  referenced  item. 

The  program  EVAL  consists  of  four  routines.  The  main  routine, 
EVAL,  processes  all  input  data  cards,  monitors  the  operations  of  tape 
reading  and  data  accumulation,  and  prints  out  summary  statistics.  The 
subroutine  EDATA  accumulates  all  RMSE  and  PTOT  performance  statistics 
according  to  directives  from  the  main  routine.  The  subroutine  TREAD 
reads  the  next  micro-report  on  the  HOS  output  tape  and  transfers  the 
message  data  to  the  calling  routine  via  the  formal  arguments  of  the  sub¬ 
routine.  The  function  SIGNAL  determines  the  value  of  the  signal  being 
tracked  at  any  given  simulation  time. 

Input  data  specifying  the  parameters  of  the  signal  being  tracked 
and  the  type  of  analysis  to  be  performed  are  required  by  EVAL.  Each 
input  data  card  must  contain  an  identifying  label  starting  in  column  1 
and  a  single  number  starting  in  column  21.  The  recognized  identifying 
labels  are: 


INTERVAL 


DELAY 

START 

END 

SIGNAL  OFFSET 
SIGNAL  FREQ1 
SIGNAL  FREQ2 
SIGNAL  FREQ 3 
SIGNAL  AMP1 
SIGNAL  AMP 2 
SIGNAL  AMP 3 
LIMIT 
TIMELINE 

The  INTERVAL  card  specifies  the  size  of  each  data  collection 
interval.  The  DELAY  card  indicates  the  length  of  time  at  the  beginning 
of  each  primary  task  execution  before  the  first  data  collection  interval 
for  that  execution  will  begin.  The  START  and  END  cards  specify  the  simula¬ 
tion  times  over  which  the  analysis  is  to  be  performed.  The  cards  that 
specify  signal  characteristics  are  self-explanatory.  Each  LIMIT  card 
(up  to  five  are  allowed)  specifies  a  PTOT  criterion  tolerance  to  be  used 
in  the  analysis.  The  TIMELINE  card  (which  needs  no  number  in  its  second 
field)  indicates  that  a  timeline  analysis  is  to  be  printed  while  the  tape 
is  being  processed.  No  timeline  report  is  printed  if  the  TIMELINE  card 
is  omitted. 

The  logic  of  the  program  is  reasonably  straightforward  and  will 
not  be  described  in  detail.  Comment  cards  in  each  routine  indicate  the 
function  performed  by  each  section  of  code.  In  order  to  assist  the 
programmer  in  interpreting  and  modifying  the  program,  the  following  list 
of  variable  descriptions  is  offered.  After  the  list  of  variable  des¬ 
criptions  is  a  full  list  of  the  EVAL  program  followed  by  a  sample 
program  output. 


IV-2 


*  Bte*  - 


VARIABLE  NAME 


DESCRIPTION 


AEND 

end  time  for  analysis 

AMP (I) 

amplitudes  of  the  three  sine  wave 
components  of  the  signal 

ASTART 

start  time  for  analysis 

BTIM(I) 

start  time  for  1^  buffer  interval  during 
a  movement 

BUFCTI(I) 

CTINT  for  the  Ith  buffer  interval 

BUFEAR(I) 

error  at  return  to  primary  for  buffer 

interval 

BUFINIT (I) 

indication  of  whether  (1)  or  not  (0) 
interval  data  accumulators  are  to  be 
initialized  after  processing  data  for 

Ith  buffer  interval 

BUFTSR(I) 

time  since  return  to  primary  task  for 
buffer  interval 

CAERRT 

cumulative  absolute  error  at  initiations 
of  primary  procedure 

CAESEC 

cumulative  absolute  error  at  initiations 
of  secondary  procedure 

CLEVEL 

value  of  " track"  when  at  a  constant 
value  between  movements 

CTINT 

time  when  current  data  collection  interval 
began 

CUECI 

cumulative  squared  error  for  current 
interval 

CUEPRI 

cumulative  squared  error  for  primary 
procedure 

CUESEC 

cumulative  squared  error  for  secondary 
procedure 

ECI 

cumulative  error  for  current  interval 

ECUM(I) 


cumulative  error  for  primary  procedure 
if  I  =  1  and  for  secondary  procedure 
if  I  »  2 


VARIABLE 
ECUM ( I ,  J ) 

ERRET 

ETIM(I) 

IINT(I) 

IMOVE 

INTEND (I) 

ITLIM(I) 

MARGIN (I) 

MOVE ALL 

MOVECI 

NAERRT 

NAESEC 

NBUF 

NCI 

NCPRI 


NAME  DESCRIPTION 


cumulative  error  for  J th  interval  after 
start  of  primary  procedure  if  I  =  1  or 
of  secondary  procedure  if  I  =  2 

actual  error  when  current  execution  of 
primary  procedure  began 

end  time  for  1^  buffer  interval 

NINT  for  Ith  buffer  interval 

1  if  a  movement  is  in  progress,  2  if  a 
movement  has  been  completed  but  data  has 
not  been  stored  because  "actual  value 
changed"  message  has  not  yet  been 
encountered,  and  0  otherwise 

1  if  the  I^h  buffer  interval  marks  the  end 
of  a  timed  interval  and  0  otherwise 

integerized  value  of  the  I th  criterion 
tolerance  (TLIM(I))  for  PTOT  determinations 

the  (first  or  second)  word  in  the 

identifier  field  of  an  input  data  card 

counter  of  all  control  movements  over 
time  span  of  analysis 

counter  of  movements  for  current  interval 

number  of  initiations  of  primary  procedure 

number  of  initiations  of  secondary  procedure 

number  of  items  in  movement  buffer 

number  of  error  observations  for  current 
interval 

number  of  observations  of  squared  error 
for  primary  procedure 


NCSEC 


number  of  observations  of  squared  error 
for  secondary  procedure 


VARIABLE  NAME 


DESCRIPTION 


NINT 


NLIMIT 

NOBS (I) 

NSEC (I) 

NTLINE 

OFFSET 

PCINT 

PRIMT 

SAERRT 

SAESEC 


Current  interval  number  after  beginning 
of  executing  procedure  (maximum  of  20) 
if  interval  data  is  being  collected  — 
value  is  positive  for  primary  procedure 
and  negative  for  secondary  procedure  — 
value  of  0  indicates  primary  procedure 
is  executing  but  no  data  should  be  col¬ 
lected  and  value  of  -30  indicates  that 
secondary  procedure  is  executing  but  no 
data  should  be  collected 

the  number  of  PTOT  criterion  limits 
specified  in  the  input  data 

number  of  observations  of  RMSE  measures 
for  I^h  interval  after  start  of  primary 
procedure 

number  of  observations  of  RMSE  measures 
for  ith  interval  after  start  of  secondary 
procedure 

number  of  lines  on  current  page  of  time¬ 
line  output 

signal  offset 

time  when  current  processing  chunk  of 
current  data  collection  interval  began 

total  time  spent  on  primary  procedure 

sum  of  squares  of  absolute  errors  at 
initiations  of  primary  procedure 

sum  of  squares  of  absolute  errors  at 
initiations  of  secondary  procedure 


SCENI(I) 


SECT 

SESEC(I) 


SF(I) 


sum  of  RMSE  measures  for  all  observations 
of  I^h  interval  after  start  of  primary 
procedure 

total  time  spent  on  secondary  procedure 

sum  of  PMSE  measures  for  all  observations 
of  I*1*1  interval  after  start  of  secondary 
procedure 

frequencies  of  the  three  sine  wave  com¬ 
ponents  of  the  signal 
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VARIABLE  NAME 


DESCRIPTION 


SMOVEI(I) 

SSCENI(I) 

SSESEC (I) 

SSMOVEI (I) 

SSTTNI (I #  J) 

SSTTSEC(I,J) 

STTNI (I ,  J) 

STTSECd,  J) 

TBMOVE 

TDEL 

TEMOVE 


sum  of  numbers  of  observed  control  move¬ 
ments  over  all  observations  of  the 
interval  after  the  start  of  the  primary 
procedure 

sum  of  squares  of  RMSE  measures  for  all 
observations  of  the  interval  after 
start  of  primary  procedure 

sura  of  squares  of  RMSE  measures  for  all 
observations  of  interval  after  start 
of  secondary  procedure 

sura  of  squares  of  numbers  of  observed 
control  movements  over  all  observations 
of  the  interval  after  the  start  of 

the  primary  procedure 

sura  of  squares  of  the  percent  times  on 
target  with  respect  to  criterion  toler¬ 
ance  J  for  all  observations  of  the  I**1 
interval  after  the  start  of  the  primary 
procedure 

sum  of  squares  of  the  percent  times  on 
target  with  respect  to  criterion  toler¬ 
ance  J  for  all  observations  of  the 
interval  after  the  start  of  the  secondary 
procedure 

sum  of  the  observed  percent  times  on 
target  with  respect  to  criterion  toler¬ 
ance  J  for  all  observations  of  the  I*-*1 
interval  after  the  start  of  the  primary 
procedure 

sum  of  the  observed  percent  times  on 
target  with  respect  to  criterion  toler¬ 
ance  J  for  all  observations  of  the  I^*1 
interval  after  the  start  of  the  secondary 
procedure 

time  when  current  movement  began 

user-specified  delay  after  beginning  of 
primary  task  when  first  data  collection 
interval  begins 

time  when  most  recent  movement  ended 
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VARIABLE  NAME 


DESCRIPTION 


TINT 

TLAST 

TLIM(I) 

TOTCI (I) 


TOTPRI  (I) 


TOTSEC (I) 


TP  RIM 


TSECO 


Tl,  T2 ,  VI,  V2, 
NN,  INTCUM 
(formal  arguments 
EDATA) 


ST,  IM,  ID,  VAL 
(formal  arguments 


user-specified  duration  of  each  data  col¬ 
lection  interval 

time  of  the  previous  micro-report  message 

the  user-specified  value  of  the  I^1 
criterion  tolerance  (in  the  order  specified 
by  the  user)  for  PTOT  determinations 

cumulative  number  of  "on-target"  observa¬ 
tions  with  respect  to  the  I^h  criterion 
tolerance  for  current  data  collection 
interval  (observations  are  regularly 
spaced  at  .01  second  intervals) 

cumulative  number  of  "on  target"  observa¬ 
tions  with  respect  to  the  I^h  criterion 
tolerance  for  all  time  on  the  primary 
procedure 

cumulative  number  of  "on-target"  observa¬ 
tions  with  respect  to  the  I^h  criterion 
tolerance  for  all  time  on  the  secondary 
procedure 

0  if  the  primary  procedure  is  not  executing 
and  otherwise  is  the  time  that  the  current 
execution  began 

0  if  the  secondary  procedure  is  not  execut¬ 
ing  and  otherwise  is  the  time  that  the 
current  execution  began 

The  track  element  moves  along  the  straight 
line  connecting  the  points  (Tl,  VI)  and 
of  (T2,  V2)  where  VI  is  the  track  value  at 

time  Tl  and  V2  is  the  track  value  at  time 
T2.  NN  is  the  interval  number  relative 
to  the  start  of  the  procedure  for  which 
data  is  to  be  accumulated.  If  INTCUM=1 
data  for  the  current  interval  should  be 
stored  and  accumlator  variables  should  be 
initiatized,  but  not  if  INTCUM=0 

Data  values  obtained  by  TREAD  from  reading 
of  TREAD)  of  next  micro-report  on  HOS  output  tape. 

ST  is  simulation  time  of  report.  IM  is 
the  code  number  of  the  message  (see  text 
above  for  interpretation  of  code) .  ID 
is  the  dictionary  pointer  referenced  by 
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VARIABLE  NAME 

ST,  IM,  ID,  VAL 
(continued) 

T 

(function  of  argument 
SIGNAL) 


DESCRIPTION 

the  report.  VAL  is  the  value  (estimated 
or  actual)  of  the  item  referenced  by  the 
report 

the  simulation  time  for  which  the  signal 
of  value  is  to  be  returned 
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